{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Testing Custom Timeouts\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequistes\n",
    "You will need\n",
    " - [Git clone of Seldon Core](https://github.com/SeldonIO/seldon-core)\n",
    " - [Helm](https://github.com/kubernetes/helm)\n",
    " - [Minikube](https://github.com/kubernetes/minikube) version v0.24.0 or greater\n",
    " - [python grpc tools](https://grpc.io/docs/quickstart/python.html)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create Cluster\n",
    "\n",
    "Start minikube and ensure custom resource validation is activated and there is 5G of memory. \n",
    "\n",
    "**2018-06-13** : At present we find the most stable version of minikube across platforms is 0.25.2 as there are issues with 0.26 and 0.27 on some systems. We also find the default VirtualBox driver can be problematic on some systems to we suggest using the [KVM2 driver](https://github.com/kubernetes/minikube/blob/master/docs/drivers.md#kvm2-driver).\n",
    "\n",
    "Your start command would then look like:\n",
    "```\n",
    "minikube start --vm-driver kvm2 --memory 4096 --feature-gates=CustomResourceValidation=true --extra-config=apiserver.Authorization.Mode=RBAC\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "namespace \"seldon\" created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl create namespace seldon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "clusterrolebinding \"kube-system-cluster-admin\" created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl create clusterrolebinding kube-system-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Install Helm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "serviceaccount \"tiller\" created\n",
      "clusterrolebinding \"tiller\" created\n",
      "$HELM_HOME has been configured at /home/clive/.helm.\n",
      "\n",
      "Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.\n",
      "\n",
      "Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.\n",
      "To prevent this, run `helm init` with the --tiller-tls-verify flag.\n",
      "For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation\n",
      "Happy Helming!\n"
     ]
    }
   ],
   "source": [
    "!kubectl -n kube-system create sa tiller\n",
    "!kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller\n",
    "!helm init --service-account tiller"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Start seldon-core"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Install the custom resource definition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME:   seldon-core-crd\n",
      "LAST DEPLOYED: Thu Aug 30 09:59:45 2018\n",
      "NAMESPACE: default\n",
      "STATUS: DEPLOYED\n",
      "\n",
      "RESOURCES:\n",
      "==> v1beta1/Deployment\n",
      "NAME                        DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE\n",
      "seldon-spartakus-volunteer  1        0        0           0          0s\n",
      "\n",
      "==> v1/ServiceAccount\n",
      "NAME                        SECRETS  AGE\n",
      "seldon-spartakus-volunteer  1        0s\n",
      "\n",
      "==> v1beta1/ClusterRole\n",
      "NAME                        AGE\n",
      "seldon-spartakus-volunteer  0s\n",
      "\n",
      "==> v1beta1/ClusterRoleBinding\n",
      "NAME                        AGE\n",
      "seldon-spartakus-volunteer  0s\n",
      "\n",
      "==> v1/ConfigMap\n",
      "NAME                     DATA  AGE\n",
      "seldon-spartakus-config  3     1s\n",
      "\n",
      "==> v1beta1/CustomResourceDefinition\n",
      "NAME                                         AGE\n",
      "seldondeployments.machinelearning.seldon.io  0s\n",
      "\n",
      "\n",
      "NOTES:\n",
      "NOTES: TODO\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!helm install ../helm-charts/seldon-core-crd --name seldon-core-crd --set usage_metrics.enabled=true"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME:   seldon-core\n",
      "LAST DEPLOYED: Thu Aug 30 09:59:52 2018\n",
      "NAMESPACE: seldon\n",
      "STATUS: DEPLOYED\n",
      "\n",
      "RESOURCES:\n",
      "==> v1/ServiceAccount\n",
      "NAME    SECRETS  AGE\n",
      "seldon  1        0s\n",
      "\n",
      "==> v1beta1/Role\n",
      "NAME          AGE\n",
      "ambassador    0s\n",
      "seldon-local  0s\n",
      "\n",
      "==> v1/RoleBinding\n",
      "NAME    AGE\n",
      "seldon  0s\n",
      "\n",
      "==> v1/Pod(related)\n",
      "NAME                                                READY  STATUS             RESTARTS  AGE\n",
      "seldon-core-ambassador-7fb4575f6b-js5lm             0/2    ContainerCreating  0         0s\n",
      "seldon-core-seldon-apiserver-68f5984b49-svn5r       0/1    ContainerCreating  0         0s\n",
      "seldon-core-seldon-cluster-manager-9fdd5d6df-jj58f  0/1    ContainerCreating  0         0s\n",
      "seldon-core-redis-575979b79b-5swz9                  0/1    ContainerCreating  0         0s\n",
      "\n",
      "==> v1beta1/ClusterRole\n",
      "NAME        AGE\n",
      "seldon-crd  0s\n",
      "\n",
      "==> v1/ClusterRoleBinding\n",
      "NAME    AGE\n",
      "seldon  0s\n",
      "\n",
      "==> v1beta1/RoleBinding\n",
      "NAME        AGE\n",
      "ambassador  0s\n",
      "\n",
      "==> v1/Service\n",
      "NAME                          TYPE       CLUSTER-IP      EXTERNAL-IP  PORT(S)                        AGE\n",
      "seldon-core-ambassador-admin  NodePort   10.99.43.159    <none>       8877:32301/TCP                 0s\n",
      "seldon-core-ambassador        NodePort   10.103.77.131   <none>       8080:31756/TCP                 0s\n",
      "seldon-core-seldon-apiserver  NodePort   10.111.157.153  <none>       8080:31229/TCP,5000:31036/TCP  0s\n",
      "seldon-core-redis             ClusterIP  10.106.34.84    <none>       6379/TCP                       0s\n",
      "\n",
      "==> v1beta1/Deployment\n",
      "NAME                                DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE\n",
      "seldon-core-ambassador              1        1        1           0          0s\n",
      "seldon-core-seldon-apiserver        1        1        1           0          0s\n",
      "seldon-core-seldon-cluster-manager  1        1        1           0          0s\n",
      "seldon-core-redis                   1        1        1           0          0s\n",
      "\n",
      "\n",
      "NOTES:\n",
      "Thank you for installing Seldon Core.\n",
      "\n",
      "Documentation can be found at https://github.com/SeldonIO/seldon-core\n",
      "\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!helm install ../helm-charts/seldon-core --name seldon-core --namespace seldon \\\n",
    "    --set ambassador.enabled=true"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check all services are running before proceeding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "deployment \"seldon-core-seldon-cluster-manager\" successfully rolled out\n",
      "deployment \"seldon-core-seldon-apiserver\" successfully rolled out\n",
      "Waiting for rollout to finish: 0 of 1 updated replicas are available...\n",
      "deployment \"seldon-core-ambassador\" successfully rolled out\n"
     ]
    }
   ],
   "source": [
    "!kubectl rollout status deploy/seldon-core-seldon-cluster-manager\n",
    "!kubectl rollout status deploy/seldon-core-seldon-apiserver\n",
    "!kubectl rollout status deploy/seldon-core-ambassador "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set up REST and gRPC methods\n",
    "\n",
    "**Ensure you port forward to API Gateway**\n",
    "\n",
    "REST:\n",
    "\n",
    "```\n",
    "kubectl port-forward $(kubectl get pods -n seldon -l app=seldon-apiserver-container-app -o jsonpath='{.items[0].metadata.name}') -n seldon 8003:8080\n",
    "```\n",
    "\n",
    "GRPC:\n",
    "\n",
    "```\n",
    "kubectl port-forward $(kubectl get pods -n seldon -l app=seldon-apiserver-container-app -o jsonpath='{.items[0].metadata.name}') -n seldon 8004:5000\n",
    "```\n",
    "\n",
    "**Ensure you port forward ambassador**:\n",
    "\n",
    "```\n",
    "kubectl port-forward $(kubectl get pods -n seldon -l service=ambassador -o jsonpath='{.items[0].metadata.name}') -n seldon 8005:8080\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Install gRPC modules for the prediction protos."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cp ../proto/prediction.proto ./proto\n",
    "!python -m grpc.tools.protoc -I. --python_out=. --grpc_python_out=. ./proto/prediction.proto"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Illustration of both REST and gRPC requests. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from requests.auth import HTTPBasicAuth\n",
    "from proto import prediction_pb2\n",
    "from proto import prediction_pb2_grpc\n",
    "import grpc\n",
    "import numpy as np\n",
    "\n",
    "AMBASSADOR_API=\"localhost:8005\"\n",
    "GATEWAY_REST=\"localhost:8003\"\n",
    "GATEWAY_GRPC=\"localhost:8004\"\n",
    "\n",
    "def get_token(oauth_key,oauth_secret):\n",
    "    payload = {'grant_type': 'client_credentials'}\n",
    "    response = requests.post(\n",
    "                \"http://\"+GATEWAY_REST+\"/oauth/token\",\n",
    "                auth=HTTPBasicAuth(oauth_key, oauth_secret),\n",
    "                data=payload)\n",
    "    print(response.text)\n",
    "    token =  response.json()[\"access_token\"]\n",
    "    return token\n",
    "\n",
    "    \n",
    "def rest_request_api_gateway(oauth_key,oauth_secret):\n",
    "    token = get_token(oauth_key,oauth_secret)\n",
    "    headers = {'Authorization': 'Bearer '+token}\n",
    "    payload = {\"data\":{\"names\":[\"a\",\"b\"],\"tensor\":{\"shape\":[2,2],\"values\":[0,0,1,1]}}}\n",
    "    response = requests.post(\n",
    "                \"http://\"+GATEWAY_REST+\"/api/v0.1/predictions\",\n",
    "                headers=headers,\n",
    "                json=payload)\n",
    "    print(response.text)\n",
    "\n",
    "def grpc_request_api_gateway(oauth_key,oauth_secret,data_size):\n",
    "    token = get_token(oauth_key,oauth_secret)\n",
    "    shape = [1,data_size]\n",
    "    arr = np.random.rand(data_size)\n",
    "    datadef = prediction_pb2.DefaultData(\n",
    "            names = [\"a\",\"b\"],\n",
    "            tensor = prediction_pb2.Tensor(\n",
    "                shape = shape,\n",
    "                values = arr\n",
    "                )\n",
    "            )\n",
    "    request = prediction_pb2.SeldonMessage(data = datadef)\n",
    "    channel = grpc.insecure_channel(GATEWAY_GRPC)\n",
    "    stub = prediction_pb2_grpc.SeldonStub(channel)\n",
    "    metadata = [('oauth_token', token)]\n",
    "    response = stub.Predict(request=request,metadata=metadata)\n",
    "    print(response)\n",
    "    \n",
    "def grpc_request_ambassador(deploymentName,data_size):\n",
    "    shape = [1,data_size]\n",
    "    arr = np.random.rand(data_size)\n",
    "    datadef = prediction_pb2.DefaultData(\n",
    "            names = [\"a\",\"b\"],\n",
    "            tensor = prediction_pb2.Tensor(\n",
    "                shape = shape,\n",
    "                values = arr\n",
    "                )\n",
    "            )\n",
    "    request = prediction_pb2.SeldonMessage(data = datadef)\n",
    "    channel = grpc.insecure_channel(AMBASSADOR_API)\n",
    "    stub = prediction_pb2_grpc.SeldonStub(channel)\n",
    "    metadata = [('seldon',deploymentName)]\n",
    "    response = stub.Predict(request=request,metadata=metadata)\n",
    "    print(response)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "    \u001b[34;01m\"apiVersion\"\u001b[39;49;00m: \u001b[33m\"machinelearning.seldon.io/v1alpha2\"\u001b[39;49;00m,\r\n",
      "    \u001b[34;01m\"kind\"\u001b[39;49;00m: \u001b[33m\"SeldonDeployment\"\u001b[39;49;00m,\r\n",
      "    \u001b[34;01m\"metadata\"\u001b[39;49;00m: {\r\n",
      "        \u001b[34;01m\"labels\"\u001b[39;49;00m: {\r\n",
      "            \u001b[34;01m\"app\"\u001b[39;49;00m: \u001b[33m\"seldon\"\u001b[39;49;00m\r\n",
      "        },\r\n",
      "        \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"seldon-deployment-example\"\u001b[39;49;00m\r\n",
      "    },\r\n",
      "    \u001b[34;01m\"spec\"\u001b[39;49;00m: {\r\n",
      "        \u001b[34;01m\"annotations\"\u001b[39;49;00m: {\r\n",
      "            \u001b[34;01m\"project_name\"\u001b[39;49;00m: \u001b[33m\"FX Market Prediction\"\u001b[39;49;00m,\r\n",
      "            \u001b[34;01m\"deployment_version\"\u001b[39;49;00m: \u001b[33m\"v1\"\u001b[39;49;00m,\r\n",
      "\t    \u001b[34;01m\"seldon.io/rest-read-timeout\"\u001b[39;49;00m:\u001b[33m\"1\"\u001b[39;49;00m\r\n",
      "        },\r\n",
      "        \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"test-deployment\"\u001b[39;49;00m,\r\n",
      "        \u001b[34;01m\"oauth_key\"\u001b[39;49;00m: \u001b[33m\"oauth-key\"\u001b[39;49;00m,\r\n",
      "        \u001b[34;01m\"oauth_secret\"\u001b[39;49;00m: \u001b[33m\"oauth-secret\"\u001b[39;49;00m,\r\n",
      "        \u001b[34;01m\"predictors\"\u001b[39;49;00m: [\r\n",
      "            {\r\n",
      "                \u001b[34;01m\"componentSpecs\"\u001b[39;49;00m: [{\r\n",
      "                    \u001b[34;01m\"spec\"\u001b[39;49;00m: {\r\n",
      "                        \u001b[34;01m\"containers\"\u001b[39;49;00m: [\r\n",
      "                            {\r\n",
      "                                \u001b[34;01m\"image\"\u001b[39;49;00m: \u001b[33m\"seldonio/mock_classifier_grpc:1.0\"\u001b[39;49;00m,\r\n",
      "                                \u001b[34;01m\"imagePullPolicy\"\u001b[39;49;00m: \u001b[33m\"IfNotPresent\"\u001b[39;49;00m,\r\n",
      "                                \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"classifier\"\u001b[39;49;00m,\r\n",
      "                                \u001b[34;01m\"resources\"\u001b[39;49;00m: {\r\n",
      "                                    \u001b[34;01m\"requests\"\u001b[39;49;00m: {\r\n",
      "                                        \u001b[34;01m\"memory\"\u001b[39;49;00m: \u001b[33m\"1Mi\"\u001b[39;49;00m\r\n",
      "                                    }\r\n",
      "                                }\r\n",
      "                            }\r\n",
      "                        ],\r\n",
      "                        \u001b[34;01m\"terminationGracePeriodSeconds\"\u001b[39;49;00m: \u001b[34m20\u001b[39;49;00m\r\n",
      "                    }\r\n",
      "                }],\r\n",
      "                \u001b[34;01m\"graph\"\u001b[39;49;00m: {\r\n",
      "                    \u001b[34;01m\"children\"\u001b[39;49;00m: [],\r\n",
      "                    \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"classifier\"\u001b[39;49;00m,\r\n",
      "                    \u001b[34;01m\"endpoint\"\u001b[39;49;00m: {\r\n",
      "\t\t\t\u001b[34;01m\"type\"\u001b[39;49;00m : \u001b[33m\"GRPC\"\u001b[39;49;00m\r\n",
      "\t\t    },\r\n",
      "                    \u001b[34;01m\"type\"\u001b[39;49;00m: \u001b[33m\"MODEL\"\u001b[39;49;00m\r\n",
      "                },\r\n",
      "                \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"fx-market-predictor\"\u001b[39;49;00m,\r\n",
      "                \u001b[34;01m\"replicas\"\u001b[39;49;00m: \u001b[34m1\u001b[39;49;00m,\r\n",
      "\t\t\u001b[34;01m\"annotations\"\u001b[39;49;00m: {\r\n",
      "\t\t    \u001b[34;01m\"predictor_version\"\u001b[39;49;00m : \u001b[33m\"v1\"\u001b[39;49;00m\r\n",
      "\t\t}\r\n",
      "            }\r\n",
      "        ]\r\n",
      "    }\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!pygmentize resources/model_short_timeouts.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Seldon Deployment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Deploy the runtime graph to kubernetes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment \"seldon-deployment-example\" created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl apply -f resources/model_short_timeouts.json -n seldon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl get seldondeployments -n seldon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!kubectl describe seldondeployments seldon-deployment-example -n seldon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the status of the SeldonDeployment. **When ready the replicasAvailable should be 1**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "map[predictorStatus:[map[replicasAvailable:1 name:test-deployment-fx-market-predictor-svc-orch replicas:1] map[name:test-deployment-fx-market-predictor-classifier-0 replicas:1 replicasAvailable:1]]]"
     ]
    }
   ],
   "source": [
    "!kubectl get seldondeployments seldon-deployment-example -o jsonpath='{.status}' -n seldon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get predictions with short and long timeouts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This next request should fail as the timeout is too short"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"access_token\":\"02843613-4954-4c75-8dd2-ab9d61fbfe16\",\"token_type\":\"bearer\",\"expires_in\":43151,\"scope\":\"read write\"}\n",
      "{\n",
      "  \"code\": 103,\n",
      "  \"info\": \"Status code: 500 Reason: \",\n",
      "  \"reason\": \"Microservice error\",\n",
      "  \"status\": \"FAILURE\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "rest_request_api_gateway('oauth-key','oauth-secret')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Send a large request which will be above the default gRPC message size and will fail."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment \"seldon-deployment-example\" deleted\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl delete -f resources/model_short_timeouts.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "    \u001b[34;01m\"apiVersion\"\u001b[39;49;00m: \u001b[33m\"machinelearning.seldon.io/v1alpha2\"\u001b[39;49;00m,\r\n",
      "    \u001b[34;01m\"kind\"\u001b[39;49;00m: \u001b[33m\"SeldonDeployment\"\u001b[39;49;00m,\r\n",
      "    \u001b[34;01m\"metadata\"\u001b[39;49;00m: {\r\n",
      "        \u001b[34;01m\"labels\"\u001b[39;49;00m: {\r\n",
      "            \u001b[34;01m\"app\"\u001b[39;49;00m: \u001b[33m\"seldon\"\u001b[39;49;00m\r\n",
      "        },\r\n",
      "        \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"seldon-deployment-example\"\u001b[39;49;00m\r\n",
      "    },\r\n",
      "    \u001b[34;01m\"spec\"\u001b[39;49;00m: {\r\n",
      "        \u001b[34;01m\"annotations\"\u001b[39;49;00m: {\r\n",
      "            \u001b[34;01m\"project_name\"\u001b[39;49;00m: \u001b[33m\"FX Market Prediction\"\u001b[39;49;00m,\r\n",
      "            \u001b[34;01m\"deployment_version\"\u001b[39;49;00m: \u001b[33m\"v1\"\u001b[39;49;00m,\r\n",
      "\t    \u001b[34;01m\"seldon.io/rest-read-timeout\"\u001b[39;49;00m:\u001b[33m\"10000\"\u001b[39;49;00m\r\n",
      "        },\r\n",
      "        \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"test-deployment\"\u001b[39;49;00m,\r\n",
      "        \u001b[34;01m\"oauth_key\"\u001b[39;49;00m: \u001b[33m\"oauth-key\"\u001b[39;49;00m,\r\n",
      "        \u001b[34;01m\"oauth_secret\"\u001b[39;49;00m: \u001b[33m\"oauth-secret\"\u001b[39;49;00m,\r\n",
      "        \u001b[34;01m\"predictors\"\u001b[39;49;00m: [\r\n",
      "            {\r\n",
      "                \u001b[34;01m\"componentSpecs\"\u001b[39;49;00m: [{\r\n",
      "                    \u001b[34;01m\"spec\"\u001b[39;49;00m: {\r\n",
      "                        \u001b[34;01m\"containers\"\u001b[39;49;00m: [\r\n",
      "                            {\r\n",
      "                                \u001b[34;01m\"image\"\u001b[39;49;00m: \u001b[33m\"seldonio/mock_classifier_rest:1.0\"\u001b[39;49;00m,\r\n",
      "                                \u001b[34;01m\"imagePullPolicy\"\u001b[39;49;00m: \u001b[33m\"IfNotPresent\"\u001b[39;49;00m,\r\n",
      "                                \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"classifier\"\u001b[39;49;00m,\r\n",
      "                                \u001b[34;01m\"resources\"\u001b[39;49;00m: {\r\n",
      "                                    \u001b[34;01m\"requests\"\u001b[39;49;00m: {\r\n",
      "                                        \u001b[34;01m\"memory\"\u001b[39;49;00m: \u001b[33m\"1Mi\"\u001b[39;49;00m\r\n",
      "                                    }\r\n",
      "                                }\r\n",
      "                            }\r\n",
      "                        ],\r\n",
      "                        \u001b[34;01m\"terminationGracePeriodSeconds\"\u001b[39;49;00m: \u001b[34m20\u001b[39;49;00m\r\n",
      "                    }\r\n",
      "                }],\r\n",
      "                \u001b[34;01m\"graph\"\u001b[39;49;00m: {\r\n",
      "                    \u001b[34;01m\"children\"\u001b[39;49;00m: [],\r\n",
      "                    \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"classifier\"\u001b[39;49;00m,\r\n",
      "                    \u001b[34;01m\"endpoint\"\u001b[39;49;00m: {\r\n",
      "\t\t\t\u001b[34;01m\"type\"\u001b[39;49;00m : \u001b[33m\"REST\"\u001b[39;49;00m\r\n",
      "\t\t    },\r\n",
      "                    \u001b[34;01m\"type\"\u001b[39;49;00m: \u001b[33m\"MODEL\"\u001b[39;49;00m\r\n",
      "                },\r\n",
      "                \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"fx-market-predictor\"\u001b[39;49;00m,\r\n",
      "                \u001b[34;01m\"replicas\"\u001b[39;49;00m: \u001b[34m1\u001b[39;49;00m,\r\n",
      "\t\t\u001b[34;01m\"annotations\"\u001b[39;49;00m: {\r\n",
      "\t\t    \u001b[34;01m\"predictor_version\"\u001b[39;49;00m : \u001b[33m\"v1\"\u001b[39;49;00m\r\n",
      "\t\t}\r\n",
      "            }\r\n",
      "        ]\r\n",
      "    }\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!pygmentize resources/model_long_timeouts.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment \"seldon-deployment-example\" created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl apply -f resources/model_long_timeouts.json -n seldon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "map[predictorStatus:[map[replicas:1 replicasAvailable:1 name:test-deployment-fx-market-predictor-svc-orch] map[replicasAvailable:1 name:test-deployment-fx-market-predictor-classifier-0 replicas:1]]]"
     ]
    }
   ],
   "source": [
    "!kubectl get seldondeployments seldon-deployment-example -o jsonpath='{.status}' -n seldon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This nrxt request should work as the timeout is much longer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"access_token\":\"02843613-4954-4c75-8dd2-ab9d61fbfe16\",\"token_type\":\"bearer\",\"expires_in\":42938,\"scope\":\"read write\"}\n",
      "{\n",
      "  \"meta\": {\n",
      "    \"puid\": \"g7u0qmuvc8tmlu9aslbgtghh4p\",\n",
      "    \"tags\": {\n",
      "    },\n",
      "    \"routing\": {\n",
      "    }\n",
      "  },\n",
      "  \"data\": {\n",
      "    \"names\": [\"proba\"],\n",
      "    \"tensor\": {\n",
      "      \"shape\": [2, 1],\n",
      "      \"values\": [0.05133579311531625, 0.12823373759251927]\n",
      "    }\n",
      "  }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "rest_request_api_gateway('oauth-key','oauth-secret')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
